Many systems enable users to connect over a network. For example, Virtual Reality (VR) systems allow users to control avatars or other virtual representations within a virtual environment. In particular, a first user at a first location may use a VR headset or other device to interact with a second user at a second location. In another example, Augmented Reality (AR) systems allow users to experience a physical world with augmented content. Specifically, a first user and a second user at a same location may view real-time images of a physical world with virtual content that is overlaid on the real-time images. In yet another example, users may communicate through a video conference system by viewing real-time images of each other. Although these systems enable users to interact with each other in a virtual manner, they suffer various setbacks. For example, VR systems generally place users within their own virtual environment. This often causes users that are located in the same physical environment to run into each other or otherwise contact each other as they move within the physical environment. Further, AR systems generally allow users that are in the same physical environment to interact but lack support for remote users. Moreover, these systems provide relatively high-level interaction that fails to mimic how humans actually communicate. For example, video conferencing systems require a user to look into a camera in order to appear to another user as making eye contact. Furthermore, since different types of systems and/or systems with multiple components are used to connect users over a network, the systems are prone to losing one or more aspects of the interaction.